Skip to content

Conversation

@PederHP
Copy link
Member

@PederHP PederHP commented Sep 1, 2025

Fixes #747

We might want to warn or give errors if StructuredContent is present when there are incompatible content blocks - ie more than one text content block. I don't think it can actually happen, but if it can it will be a spec violation. Clients should not get different output in unstructured Content if StructuredContent is provided.

@PederHP PederHP changed the title Fix structured output backwards compatibility for string literals Fix structured output backwards compatibility for string results Sep 1, 2025
// If there is structuredOutput we must stringify it, as the MCP specification requires Content to be for backwards compatibility
// but otherwise not differ
Content = [new TextContentBlock { Text = structuredContent == null ? text :
JsonSerializer.Serialize(structuredContent, AIFunction.JsonSerializerOptions.GetTypeInfo(typeof(object)))}],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not just resolve the type of structuredContent itself and skip one degree of indirection?

Suggested change
JsonSerializer.Serialize(structuredContent, AIFunction.JsonSerializerOptions.GetTypeInfo(typeof(object)))}],
JsonSerializer.Serialize(structuredContent, AIFunction.JsonSerializerOptions.GetTypeInfo<JsonNode>()))}],

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not just resolve the type of structuredContent itself and skip one degree of indirection?

Suggested change
JsonSerializer.Serialize(structuredContent, AIFunction.JsonSerializerOptions.GetTypeInfo(typeof(object)))}],
JsonSerializer.Serialize(structuredContent, AIFunction.JsonSerializerOptions.GetTypeInfo<JsonNode>()))}],

Good point, but this goes for the default switch case too then?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I suppose we can leave it for consistency.

@PederHP
Copy link
Member Author

PederHP commented Sep 4, 2025

Should an error be thrown on the AIContent related return types? If we go by the intent expressed by dsp and others, and Content SHOULD be present for backwards compatibility, but (the part missing right now) MUST not differ from StructuredContent if it is - then there is no possible way to do with these.

It would be a breaking change, so probably best to wait with that until the spec is actually updated. Just wanted to mention that as pointed out in the issue, this is only a fix for the string case. I think that's the one most likely to cause trouble on actual production servers, so I still think it's worth fixing this one in isolation. Setting the attribute property to true for an AIContent or CallToolResult returning tool is a really exotic edge-case (to be fixed with the spec change imo - at the same time as any introduced schema validation requirements).

@Izelude
Copy link

Izelude commented Oct 31, 2025

For consideration:
Requiring the Content to match the structuredContent would be an issue if trying to target OpenAIs new Apps SDK.
There structuredContent and Content explicitly do not have to match the same schema.

https://developers.openai.com/apps-sdk/build/mcp-server

a sample response from their "Structure the data your tool returns" paragraph:

return {
      structuredContent: {
        columns: board.columns.map((column) => ({
          id: column.id,
          title: column.title,
          tasks: column.tasks.slice(0, 5) // keep payload concise for the model
        }))
      },
      content: [{ type: "text", text: "Here's your latest board. Drag cards in the component to update status." }],
      _meta: {
        tasksById: board.tasksById, // full task map for the component only
        lastSyncedAt: board.lastSyncedAt
      }
    };

@PederHP
Copy link
Member Author

PederHP commented Oct 31, 2025

For consideration:
Requiring the Content to match the structuredContent would be an issue if trying to target OpenAIs new Apps SDK.
There structuredContent and Content explicitly do not have to match the same schema.

https://developers.openai.com/apps-sdk/build/mcp-server

a sample response from their "Structure the data your tool returns" paragraph:

return {
      structuredContent: {
        columns: board.columns.map((column) => ({
          id: column.id,
          title: column.title,
          tasks: column.tasks.slice(0, 5) // keep payload concise for the model
        }))
      },
      content: [{ type: "text", text: "Here's your latest board. Drag cards in the component to update status." }],
      _meta: {
        tasksById: board.tasksById, // full task map for the component only
        lastSyncedAt: board.lastSyncedAt
      }
    };

There is some discussion around this at the protocol governance level. I expect we'll see a change where they do not have to be the same.

The current requirements of the Apps SDK will cause some MCP SDKs to warn/fail, and isn't fully spec compliant, even if one can argue the spec doesn't forbid the discrepancy, this is because the assumption was they either matched or only one was provided.

Another interesting aspect is that the Apps SDK sends the unstructured text result to the model and uses the structured results for host specific processing. Implementing something similar on the client host in the C# SDK is awkward, as the convenience MEAI tool mappers assume the structured output should be sent to the model.

So the change in the spec should arguably also be accompanied by a change to this behavior to prioritize the unstructured output if supplied. Or at least make it easily configurable.

@Izelude
Copy link

Izelude commented Oct 31, 2025

Another interesting aspect is that the Apps SDK sends the unstructured text result to the model and uses the structured results for host specific processing. Implementing something similar on the client host in the c# SDK is awkward, as the convenience MEAI tool mappers assume the structured output should be sent to the model.

From what i understand from the OpenAI Apps SDK docs is that both the structured AND the unstructured content are provided to the model:
Your component receives all three fields, but only structuredContent and content are visible to the model.

@PederHP
Copy link
Member Author

PederHP commented Oct 31, 2025

Another interesting aspect is that the Apps SDK sends the unstructured text result to the model and uses the structured results for host specific processing. Implementing something similar on the client host in the c# SDK is awkward, as the convenience MEAI tool mappers assume the structured output should be sent to the model.

From what i understand from the OpenAI Apps SDK docs is that both the structured AND the unstructured content are provided to the model: Your component receives all three fields, but only structuredContent and content are visible to the model.

Ah yes - but I am not sure all of that is placed in the actual tool response content part. Either way, we're now in a situation where it is client dependent which fields go where and how, which complicated things a lot when one wants to write servers that are generic and work with multiple clients.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

StructuredContent vs Content Mismatch

3 participants